Skip to content

feat(integrations): map reasoning effort and context size for Copilot, Codex, and Claude Code - #5249

Open
sukanth wants to merge 8 commits into
github:mainfrom
sukanth:feat/4916-dispatch-settings-integrations
Open

sukanth wants to merge 8 commits into
github:mainfrom
sukanth:feat/4916-dispatch-settings-integrations

Conversation

@sukanth

@sukanth sukanth commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Description

Closes #4916.

Builds on #5248, which added the shared per-step dispatch-settings contract and is now merged. This branch is rebased onto main and contains only the eight commits below. This split follows Integration pull requests.

This PR maps the workflow step settings reasoning_effort and context_size to the native CLI arguments of three integrations. It has one commit per integration, one docs commit, and four review follow-ups.

Integration reasoning_effort context_size Validation
GitHub Copilot --reasoning-effort <v> --context <v> reasoning_effort must be none, minimal, low, medium, high, xhigh, or max, and context_size must be default or long_context: the values Copilot CLI 1.0.106 accepts. Copilot CLI repeats a rejected value in its error, so Spec Kit rejects it first.
Codex CLI -c model_reasoning_effort="<v>" -c model_context_window=<n> context_size must be a positive whole number with no leading zeros, at most 2**63 - 1, the range Codex's i64 config accepts.
Claude Code --effort <v> Not mapped. The step fails with a hint to choose a long-context model. reasoning_effort must be low, medium, high, xhigh, or max. Claude Code 2.1.318 only warns about other values and runs at its default effort.
  • Copilot:
    • Both receiving methods declare and translate the settings: build_exec_args() for prompt steps, and the overridden dispatch_command() for command steps.
    • One private helper builds the flags for both argv paths, in both the skills and commands layouts.
    • The flags go after --model and before --output-format.
    • Spec Kit checks each value against Copilot CLI 1.0.106's list before dispatch. The error names the accepted values and SPECKIT_INTEGRATION_COPILOT_EXTRA_ARGS for newer ones.
  • Codex: the -c overrides go after --model. A value is embedded as a TOML string only after it passes the host's value check, so it cannot contain quotes, backslashes, or control characters.
  • Claude Code:
    • ClaudeIntegration now owns build_exec_args(). It calls super() without the settings, which validates the runtime configuration once and builds the base argv. It then runs the gate and appends --effort.
    • With both settings unset, it returns super()'s argv unchanged.
    • The error for an unknown level names an escape hatch for levels added in newer Claude Code releases: SPECKIT_INTEGRATION_CLAUDE_EXTRA_ARGS="--effort <value>".
  • Rejected values are not repeated. The Copilot errors, the Codex context_size error, and the Claude Code reasoning_effort error state the rule but not the value. A value can come from captured agent output, which Spec Kit does not print to the terminal or write to log.jsonl. Codex's reasoning_effort stays open because Codex CLI 0.165.0 accepts any value a model advertises. The Codex guide therefore advises setting it from a literal or a workflow input, not from captured output.
  • Unset settings leave argv unchanged. With both settings unset, every integration builds byte-identical argv. Before editing, a 144-case argv snapshot was taken; afterwards the output was compared to it with cmp. The snapshot covers:
    • all three integrations;
    • build_exec_args and dispatch_command;
    • Copilot's skills and commands layouts;
    • --yolo on and off;
    • EXTRA_ARGS set and unset;
    • model set and unset;
    • output_json, streaming, and bounded capture.
  • Docs:
    • Each of the three agent guides gets a "Workflow Dispatch Settings" section.
    • docs/workflow-steps/types.md gets a support table and the Claude example line. Its shared example now runs the step with a numeric context_size on Codex, the only built-in integration that takes a token count.
    • docs/integrations/cli.md lists the built-in support.

Testing

The agent described in the AI disclosure ran all of the testing below. Nothing was run by hand. The commands ran against this PR's head commit, d5fcab0f, in a clean worktree.

  • Tested locally with uv run specify --help: exit 0.
  • Ran existing tests with uv sync && uv run pytest, in an equivalent .venv. Results are below.
  • Tested with a sample project. See "Manual test results".
Command Result
uvx ruff@0.15.0 check src tests All checks passed
npx markdownlint-cli2@0.23.2 on the five changed docs 0 issues; all relative links and anchors resolve
.venv/bin/python -m pytest tests/workflows tests/test_workflows.py tests/integrations tests/specify_cli/integrations tests/test_agent_config_consistency.py -q -n 6 6353 passed, 8 skipped
New tests only (-k DispatchSettings in the three integration test files) 94 passed: 39 Copilot, 26 Codex, 29 Claude Code
.venv/bin/python -m pytest -q -n 6 (full suite), then a serial re-run of every failure Under -n 6: 14387 passed, 369 failed, 134 skipped. All 369 are in 14 test files this PR does not change: shell-script and parity suites that fail under parallel load on this machine. A serial re-run of those 369 passed 368. The remaining test is the environment-specific test_non_windows_uv_tool_dry_run_still_previews_installer, which also fails on main (a60bd9ba) here.

Before and after. On f32b277b, the commit that completes the three mappings and their docs, the new integration tests ran twice against the contract that #5248 added to main: once with the three integration modules from main, and once with this PR's modules. For the first run, the Claude module from main got only the EFFORT_LEVELS constant, so that the test module imports.

  • Before: 63 failed, 5 passed. The five pass by design: a Copilot ownership negative (a legacy builder subclass rejects the prompt path), the injected Claude vocabulary constant, Claude's unset-argv invariant, the order of Claude's runtime-configuration and setting errors, and the existing EXTRA_ARGS escape hatch.
  • After: 68 passed.

Review fixes.

  • b3116ff7 adds six tests that feed captured output into the Codex context_size and the Claude Code reasoning_effort, both directly and through command and prompt steps. The output passes the host's value check. All six fail on f32b277b, whose errors repeated the value, and pass on b3116ff7.
  • 0f2e4971 adds 21 tests.
    • The 10 negative ones fail on b3116ff7, with only the two value lists added so that the test module imports. Unknown values reached Copilot CLI, and an engine run started it with a captured secret in its argv.
    • The other 11 pass by design: every accepted value, the list constants, and the EXTRA_ARGS escape hatch.
    • All 21 pass on 0f2e4971.

Test selection reasoning

Changed file Affects Test Why
src/specify_cli/integrations/copilot/__init__.py specify workflow run command and prompt steps with Copilot T1, T2, T4 Copilot argv paths and value checks
src/specify_cli/integrations/codex/__init__.py, claude/__init__.py specify workflow run with Codex or Claude Code Unit tests, T3, and parser evidence Neither CLI is authenticated on the test machine, so T3 uses a stand-in executable
docs/workflow-steps/types.md The documented example workflow T5 The example must run whether the workflow uses GitHub Copilot, Claude Code, or Codex
Other docs/** none none Documentation only

Manual test results

Agent: GitHub Copilot CLI 1.0.106, run by specify workflow run through an argv-logging probe that appends --version. The real CLI parses Spec Kit's argv without making a model call. Claude Code 2.1.318 and Codex CLI 0.165.0 were checked at the parser level only. OS/Shell: macOS / zsh

Command tested Notes
T1 specify workflow run with three steps: command step plan (reasoning_effort: "{{ inputs.effort }}", context_size: long_context); prompt step review (inherits workflow.reasoning_effort: medium); command step tasks (opts out with reasoning_effort: "{{ inputs.tasks_effort }}", an input that defaults to ""; a literal "" would inherit the default) Passed. The Copilot CLI received --reasoning-effort low --context long_context, then --reasoning-effort medium, then no new flags. All three steps completed with exit 0. The run state records each step's values: low/long_context, medium/null, and null/null.
T2 specify workflow run with reasoning_effort: turbo and capture: true Passed. The step failed before Copilot CLI started, with Spec Kit's error listing the accepted levels and the SPECKIT_INTEGRATION_COPILOT_EXTRA_ARGS escape hatch. Before 0f2e4971, Copilot CLI rejected the value itself and repeated it in its error.
T3 specify workflow run in which a capture: true prompt step's output (FAKEsecret0123456789, printed by a stand-in executable) feeds the next step's context_size (Codex) or reasoning_effort (Claude Code) Passed. Each run failed at the second step before the CLI started. The captured text is not in the step error, log.jsonl, or the console output. On f32b277b it was in all three.
T4 specify workflow run in which a capture: true step's output (FAKEsecret0123456789, printed by a stand-in executable) feeds the reasoning_effort of a Copilot step that does not capture Passed. The step failed before Copilot CLI started. The captured text is not in the console output or log.jsonl. On b3116ff7, Copilot CLI printed error: invalid value 'FAKEsecret0123456789' for '--reasoning-effort <level>' to the terminal.
T5 The example from "Reasoning Effort and Context Size" in docs/workflow-steps/types.md, run by specify workflow run with copilot, claude, or codex as the workflow integration, through argv-logging stand-in executables Passed for all three. plan ran on Codex with -c model_reasoning_effort="high" -c model_context_window=200000, and review ran on the workflow's integration with neither setting. On 64329bbc, plan failed on GitHub Copilot, which accepts only default or long_context, and on Claude Code, which does not support context_size.
Claude Code --effort and Codex -c overrides claude -p … --effort bogus only warns, which is why Spec Kit validates the level. codex exec -c 'model_reasoning_effort="high"' reports reasoning effort: high. A non-integer model_context_window is rejected (expected i64).

AI Disclosure

  • I did not use AI assistance for this contribution
  • I did use AI assistance (fill in the disclosure below)

AI disclosure: Implemented with the GitHub Copilot app in agent mode, running autonomously on behalf of @sukanth, who chose the issue and directed the work. An orchestrator and four implementation workers, all Claude Opus 5.5 at maximum reasoning effort with long context, wrote the change. Before implementation, a panel of five models reviewed and approved the plan: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol, and GPT-5.6 Sol at maximum reasoning effort, and Gemini 4 Argon at high reasoning effort, all with long context. The same panel reviewed the code, and the agent fixed the findings it accepted. Extent: AI generated the code, tests, docs, and this description, and the agent ran all of the testing. No human reviewed the change line by line before submission. AI-generated follow-up commits and review replies will carry their own disclosure. The review follow-ups (b3116ff7, 0f2e4971, 64329bbc, and d5fcab0f), the rebase onto main after #5248 merged, and this description's re-runs were done by the Agent merge session for this PR, also Claude Opus 5.5 at maximum reasoning effort, with no line-by-line human review.

@sukanth
sukanth requested a review from mnriem as a code owner November 5, 2026 08:28
Copilot AI auto review requested due to automatic review settings November 5, 2026 08:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Validation diagnostics can expose secret-bearing captured output in step errors and log.jsonl, violating the capture guarantee and requiring value-free diagnostics with regression coverage.

1 open issue
What changed in this PR

Adds per-step reasoning effort and context size to Spec Kit workflow dispatch, including the shared contract from prerequisite #5248 and native mappings addressing #4916.

Changes:

  • Adds workflow defaults, expression resolution, validation, and run/resume recording for both settings.
  • Maps both settings for Copilot and Codex, and reasoning effort for Claude Code, while preserving arguments when unset.
  • Adds contract, adapter, and integration tests, plus documentation of support and limitations.
File Description
tests/​workflows/​test_dispatch_settings.py Tests resolution, validation, dispatch, and resume behavior.
tests/​specify_cli/​integrations/​test_installed_adapters.py Tests installed-adapter support and legacy compatibility.
tests/​integrations/​test_integration_copilot.py Tests Copilot flags across both layouts and dispatch paths.
tests/​integrations/​test_integration_codex.py Tests Codex overrides and token-count validation.
tests/​integrations/​test_integration_claude.py Tests Claude effort levels and unsupported context settings.
tests/​integrations/​test_base.py Tests shared support declarations and validation gates.
src/​specify_cli/​workflows/​step/​prompt/​__init__.py Resolves and forwards prompt-step settings.
src/​specify_cli/​workflows/​step/​command/​__init__.py Resolves and forwards command-step settings.
src/​specify_cli/​workflows/​step/​_dispatch_settings.py Shares resolution, validation, and error attribution.
src/​specify_cli/​workflows/​engine.py Validates defaults and supplies them during run/resume.
src/​specify_cli/​workflows/​base.py Appends dispatch defaults to StepContext.
src/​specify_cli/​workflows/​_execution.py Gives called workflows their own dispatch defaults.
src/​specify_cli/​integrations/​copilot/​__init__.py Maps settings to native Copilot flags.
src/​specify_cli/​integrations/​codex/​__init__.py Maps TOML overrides and validates context counts.
src/​specify_cli/​integrations/​claude/​__init__.py Maps effort and rejects unsupported settings.
src/​specify_cli/​integrations/​base.py Defines the shared opt-in dispatch contract.
docs/​workflows/​api.md Documents workflow defaults and options semantics.
docs/​workflow-steps/​types.md Explains settings, support, recording, and compatibility.
docs/​workflow-steps/​api.md Documents new context fields.
docs/​integrations/​copilot.md Documents Copilot flags and accepted values.
docs/​integrations/​codex.md Documents Codex overrides and context semantics.
docs/​integrations/​cli.md Summarizes shared dispatch support.
docs/​integrations/​claude.md Documents effort validation and context limitations.
design/​integration.md Defines adapter ownership and opt-in requirements.

Review effort: Balanced (auto)

Comment thread src/specify_cli/integrations/base.py Resolved
@sukanth
sukanth force-pushed the feat/4916-dispatch-settings-integrations branch from e3bd6a0 to 3d58712 Compare November 5, 2026 09:07
Copilot AI auto review requested due to automatic review settings November 5, 2026 09:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the per-step reasoning-effort and context-size changes.

1 open issue

Review effort: Balanced (auto)

@sukanth

sukanth commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Review round 1 (Copilot code review of e3bd6a0d)

This branch is rebased onto #5248's new head, 6a8ae513, and the head is now 3d587126. The rebase changed only the parents of the integration and docs commits.

Changes:

  • 6a8ae513 in feat(workflows): add per-step reasoning effort and context size settings #5248, fix(workflows): keep rejected dispatch setting values out of errors: an error about a malformed dispatch-setting value reports only its type and length. design/integration.md now tells integrations to keep the value out of validate_dispatch_setting() errors.
  • 3d587126, fix(integrations): omit rejected values from Claude and Codex errors: the Codex context_size error and the Claude Code reasoning_effort error state the rule without the value.

Evidence on 3d587126:

  • The six tests added in 3d587126 failed on the rebased cc6ce181 and pass now. feat(workflows): add per-step reasoning effort and context size settings #5248's description covers 6a8ae513.
  • T3 in the description is a specify workflow run in which captured output feeds the next step's setting. On e3bd6a0d, the captured text reached the step error, log.jsonl, and the console. On 3d587126, it reaches none of them.
  • ruff and markdownlint are clean. The focused suites passed, and the full suite has no failures outside main's own failure set. T1 and T2 were re-run with the same results. The description's Testing section has the details.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote the fix, the tests, and this summary, and ran all of the testing; no human reviewed them line by line.

@mnriem mnriem added the triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review label Nov 5, 2026
@mnriem

mnriem commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

@mnriem mnriem added the author-awaiting Waiting on author response label Nov 5, 2026
@sukanth

sukanth commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Please address Copilot feedback

The Copilot feedback on this PR is the one thread on base.py, and it is addressed. 6a8ae513 (in #5248) and 3d587126 keep rejected dispatch-setting values out of step errors, with regression tests, and the thread has a reply describing the fix. Copilot's re-review of 3d587126 reports no active issues, and CI passes on that commit. The author will resolve the thread; per CONTRIBUTING, the agent does not resolve reviewer conversations.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote this reply; no human reviewed it before posting.

@mnriem

mnriem commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

The Copilot feedback is addressed, and that review conversation is resolved. This PR remains blocked pending the merge of #5248; once that prerequisite merges, please rebase this branch to remove the prerequisite commits so the remaining changes can receive final review before merging.

AI disclosure: Posted on behalf of @mnriem by the GitHub Copilot app using GPT-6.1 Sol in interactive agent mode. AI drafted and posted this comment at the maintainer’s direction; model settings are not exposed to the agent.

@sukanth
sukanth force-pushed the feat/4916-dispatch-settings-integrations branch from 3d58712 to 544bc60 Compare November 6, 2026 19:33
Copilot AI auto review requested due to automatic review settings November 6, 2026 19:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the per-step reasoning-effort and context-size changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced (auto)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Captured secrets can pass unchanged into Copilot dispatch settings and be echoed in native errors to terminal or CI logs.

1 open issue

Review effort: Max (auto)

Copilot AI auto review requested due to automatic review settings November 6, 2026 21:25
@sukanth

sukanth commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Review round 2 (Copilot code review of 544bc60b)

The head is now 3ba70dab.

Changes:

  • 932084de, fix(copilot): validate dispatch setting values before dispatch:
    • Spec Kit accepts only the values Copilot CLI 1.0.106 lists: none, minimal, low, medium, high, xhigh, and max for reasoning_effort, and default and long_context for context_size.
    • Any other value fails the step before the CLI starts, so Copilot CLI never repeats it.
    • The error names the accepted values and the SPECKIT_INTEGRATION_COPILOT_EXTRA_ARGS escape hatch, but not the value.
    • This replaces the pass-through behavior that the Copilot guide and this description used to describe.
  • 3ba70dab, docs(codex): keep captured output out of reasoning effort: Codex's reasoning_effort stays open.
    • Codex CLI 0.165.0 accepts any non-empty value that the model advertises. See ReasoningEffort::Custom in codex-rs/protocol/src/openai_models.rs at rust-v0.165.0. A fixed list would therefore reject valid values.
    • Codex displays the effort it receives, so its guide now advises setting it from a literal or a workflow input, not from captured output.

Evidence:

  • 932084de adds 21 tests. Its 10 negative tests failed on 544bc60b, where an engine run started Copilot CLI with a captured secret in its argv. All 21 pass on 3ba70dab.
  • T4 in the description: on 544bc60b, a captured secret in the reasoning_effort of a Copilot step that does not capture reached the terminal through Copilot CLI's error. On 3ba70dab, the CLI does not start, and the secret is in neither the console nor log.jsonl. T2 now fails before dispatch with Spec Kit's error.
  • ruff and markdownlint are clean. The focused and full suite results are in the description's Testing section.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote the changes, the tests, and this summary, and ran all of the testing; no human reviewed them line by line.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because an unpublished critical finding indicates that forwarded reasoning-effort values may expose secrets in terminal or CI logs.

No open issues 🎉

1 resolved since last review

Review effort: Max (auto)

@sukanth

sukanth commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Please address Copilot feedback

Addressed .

Translate the reasoning_effort and context_size workflow step settings
to --reasoning-effort and --context on both Copilot argv paths: prompt
steps through build_exec_args() and command steps through the
overridden dispatch_command(). The flags follow --model. Copilot CLI
validates the values itself.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sukanth Gunda and others added 6 commits November 9, 2026 07:42
Translate reasoning_effort to -c model_reasoning_effort="<value>" and
context_size to -c model_context_window=<n> after --model.
context_size must be a positive whole number no larger than the i64
range that Codex accepts.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Claude Code only warns about an unknown --effort value and then runs
at its default effort, so Spec Kit checks the value against Claude's
documented levels (low, medium, high, xhigh, max) before dispatch.
The error names SPECKIT_INTEGRATION_CLAUDE_EXTRA_ARGS as an escape
hatch for newer levels. context_size is not mapped; its error points
to choosing a long-context model.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add the support table for GitHub Copilot, Codex CLI, and Claude Code
to the workflow step reference and the shared CLI dispatch guide.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A dispatch setting can resolve from captured agent output, and a step
error is printed and written to log.jsonl, where captured text must
never appear. Claude Code's reasoning_effort check and Codex's
context_size check repeated the rejected value, so captured text that
passes the host's value pattern reached both. The errors now state the
setting and its rule without the value.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot CLI rejects an unknown --reasoning-effort level or --context tier
with an error that repeats the value. A setting can resolve from captured
agent output, so a step that does not capture could print captured text
to the terminal. Spec Kit now accepts only the values that Copilot CLI
1.0.106 lists, and fails the step before the CLI starts. The error names
the accepted values and the SPECKIT_INTEGRATION_COPILOT_EXTRA_ARGS escape
hatch, but not the value.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codex displays the reasoning effort it is given and accepts any value a
model advertises, so Spec Kit cannot check it against a fixed list. The
guide now advises setting it from a literal or a workflow input rather
than from captured output.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sukanth
sukanth force-pushed the feat/4916-dispatch-settings-integrations branch from 3ba70da to 64329bb Compare November 9, 2026 13:45

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The numeric context_size example fails when it inherits Copilot or Claude Code because neither integration supports that value.

No open issues 🎉

Review effort: Max (auto)

Copilot AI auto review requested due to automatic review settings November 9, 2026 13:51
The shared example sets context_size: "200000" on a step that inherits
the workflow's integration. Of the built-in integrations, only Codex
takes a token count, so the step failed when the workflow ran on GitHub
Copilot or Claude Code. The example now dispatches that step to Codex.

Part of github#4916.

Assisted-by: GitHub Copilot (model: Claude Opus 5.5, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Parser-only checks for Claude and Codex leave runtime CLI compatibility unverified, while unmerged prerequisite PR #5248 requires a maintainer to enforce merge order.

No open issues 🎉

Review effort: Max (auto)

Copilot AI auto review requested due to automatic review settings November 9, 2026 14:02
@sukanth

sukanth commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Review round 3 (Copilot reviews of 3ba70dab and 64329bbc)

once that prerequisite merges, please rebase this branch to remove the prerequisite commits so the remaining changes can receive final review before merging.

Done. #5248 merged as a60bd9ba, whose diff matches #5248's branch exactly, and this branch is rebased onto it. Only this PR's commits remain. git range-diff shows the seven earlier commits unchanged apart from their base. The head is now d5fcab0f.

Changes:

  • d5fcab0f, docs(workflows): run the context size example on Codex: the example under "Reasoning Effort and Context Size" set context_size: "200000" on a step that inherits the workflow's integration. It therefore failed on GitHub Copilot, which accepts default or long_context, and on Claude Code, which does not support context_size. The example now dispatches that step to Codex.

Not changed:

  • The 21:44Z review's unpublished finding about forwarded reasoning-effort values concerns Codex, whose reasoning_effort stays open. Round 2 explains why.
  • The 14:02Z review notes that Claude Code and Codex were checked at the parser level only. Neither CLI is authenticated on the test machine, as the description's Testing section states. Its merge-order note predates this rebase.

Evidence on d5fcab0f:

  • T5 in the description runs the documented example with GitHub Copilot, Claude Code, and Codex as the workflow integration. On 64329bbc, plan failed on Copilot and Claude Code. On d5fcab0f, all three runs complete.
  • After the rebase, I re-ran the focused suite, the regression tests for both earlier review fixes, and T1–T4, with the same results. The full suite has no failures in files this PR changes. ruff and markdownlint are clean.

Posted on behalf of @sukanth by the GitHub Copilot app (agent mode, Agent merge, autonomous) with Claude Opus 5.5 at maximum reasoning effort. AI wrote the rebase, the docs fix, and this summary, and ran all of the testing; no human reviewed them line by line.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because captured reasoning_effort values can expose secrets through Codex output to terminals or CI logs when capture is disabled.

No open issues 🎉

Review effort: Max (auto)

@mnriem
mnriem requested an auto review from Copilot November 9, 2026 18:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author-awaiting Waiting on author response triage-nice-to-have Verdict: evidence-backed fix or greenlit feature — land after review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Per-step reasoning effort and context size for command steps, passed through like model

3 participants